Virtual links are generalizations of classical links that can be represented by links embedded in a ``thickened'' surface $\Sigma\times I$, product of a Riemann surface of genus $h$ with an interval. In this paper, we show that virtual alternating links and tangles are naturally associated with the $1/N^2$ expansion of an integral over $N\times N$ complex matrices. We suggest that it is sufficient to count the equivalence classes of these diagrams modulo ordinary (planar) flypes. To test this hypothesis, we use an algorithm coding the corresponding Feynman diagrams by means of permutations that generates virtual diagrams up to 6 crossings and computes various invariants. Under this hypothesis, we use known results on matrix integrals to get the generating functions of virtual alternating tangles of genus 1 to 5 up to order 10 (i.e.\ 10 real crossings). The asymptotic behavior for $n$ large of the numbers of links and tangles of genus $h$ and with $n$ crossings is also computed for $h=1,2,3$ and conjectured for general $h$.